Section: Scientific Foundations

Software Engineering for Activity Recognition

Participants : Sabine Moisan, Annie Ressouche, Jean-Paul Rigault, François Brémond.

Introduction

The aim of this research axis is to build general solutions and tools to develop systems dedicated to activity recognition. For this, we rely on state-of-the art Software Engineering practices to ensure both sound design and easy use, providing genericity, modularity, adaptability, reusability, extensibility, dependability, and maintainability.

This year we focused on four aspects: the definition of a joint software platform with The Orion/Pulsar spin off KEENEO, the study of model-driven engineering approaches to facilitate platform usage, the extension of behavioral models, and formal verification techniques to design dependable systems.

Platform for Activity Recognition

Figure 1. Global Architecture of an Activity Recognition Platform.

In the former project team Orion, we have developed two platforms, one (VSIP), a library of real-time video understanding modules and another one, Lama [13] , a software platform enabling to design not only knowledge bases, but also inference engines, and additional tools. lama offers toolkits to build and to adapt all the software elements that compose a knowledge-based system or a cognitive system.

Pulsar will continue to study generic systems and object-oriented frameworks to elaborate a methodology for the design of activity recognition systems. We want to broaden the approach that led to Lama and to apply it to the other components of the activity recognition platform, in particular to the image processing ones. We also wish to contribute to set up, in the long term, a complete software engineering methodology to develop activity recognition systems. This methodology should be based on model engineering and formal techniques.

To this end, Pulsar plans to develop a new platform (see Figure 1 ) which integrates all the necessary modules for the creation of real-time activity recognition systems. Software generators provide designers with perception, software engineering and knowledge frameworks. Designers will use these frameworks to create both dedicated activity recognition engines and interactive tools. The perception and evaluation interactive tools enable a perception expert to create a dedicated perception library. The knowledge acquisition, learning and evaluation tools enable a domain expert to create a new dedicated knowledge base.

We plan to work in the following three research directions: models (adapted to the activity recognition domain), platform architecture (to cope with deployment constraints such as real time or distribution), and system safeness (to generate dependable systems). For all these tasks we shall follow state-of-the-art Software Engineering practice and, when needed, we shall attempt to set up new ones.

The new platform should be easy to use. We should thus define and implement tools to support modeling, design, verification inside the framework. Another important issue deals with user graphical interfaces. It should be possible to plug existing (domain or application dependent) graphical interfaces into the platform. This requires defining a generic layer to accommodate various sorts of interfaces. This is clearly a medium/long term goal, in its full generality at least.

Software Modeling for Activity Recognition

Developing integrated platforms such as SUP is a current trend in video surveillance. It is also a challenge since these platforms are complex and difficult to understand, to use, to validate, and to maintain. The situation gets worse when considering the huge number of choices and options, both at the application and platform levels. Dealing with such a variability requires formal modeling approaches for the task specification as well as for the software component description.

Model Driven Engineering (MDE)  [73] is a recent line of research that appears as an excellent candidate to support this modeling effort while providing means to make models operational and even executable. Our goal is to explore and enrich MDE techniques and model transformations to support the development of product lines for domains presenting multiple variability factors such as video surveillance.

More specifically, we first wish to use MDE to assist video-surveillance designers as well as software developers in configuring a video system before and during its deployment. A second challenge is to exploit models at runtime to cope with the dynamic aspects of video surveillance systems. Both approaches rely on software models, in our case feature diagrams  [65] , [58] , [74] , [54] and model transformations.

On the MDE side, we wish to identify the limits of current techniques when applied to real scale complex tasks. On the video surveillance side, the trend is toward integrated software platforms, which requires formal modeling approaches for the task specification as well as for the software component description.

This MDE approach is complementary to the Program Supervision one, which has been studied by Orion for a long time [12] . Program Supervision focuses on programs, their models and the control of their execution. MDE also covers task specification and transformations to a design and implementation.

Behavioral Models for Activity Recognition

Pursuing the work done in Orion, we need to consider other models to express knowledge about activities, their actors, their relations, and their behaviors.

The evolution toward activity recognition requires various theoretical studies. The incorporation of a model of time, both physical and logical, is mandatory to deal with temporal activity recognition especially in real time. A fundamental concern is to define an abstract model of scenarios to describe and recognize activities. Finally, handling uncertainty is a major theme of Pulsar and we want to introduce it into our platform; this requires deep theoretical studies and is a long term goal.

Safeness of Systems for Activity Recognition

Another aim is to build dependable systems. Since traditional testing is not sufficient, it is important to rely on formal verification techniques and to adapt them to our component models.

In most activity recognition systems, safeness is a crucial issue. It is a very general notion dealing with person and goods protection, respect of privacy, or even legal constraints. However, when designing software systems it will end up with software security. In Orion, we already provided toolkits to ensure validation and verification of systems built with Lama . First, we offered a knowledge base verification toolkit, allowing to verify the consistency and the completeness of a base as well as the adequacy of the knowledge with regard to the way an engine is going to use it. Second, we also provided an engine verification toolkit that relies on model-checking techniques to verify that the Blocks library has been used in a safe way during knowledge based system engine designs.

Generation of dependable systems for activity recognition is an important challenge. System validation really is a crucial phase in any development cycle. Partial validation by tests, although required in the first phase of validation, appears to be too weak for the system to be completely trusted. An exhaustive approach of validation using formal methods is clearly needed. Formal methods help to produce a code that has been formally proved and the size and frequency of which can be estimated. Consistently with our component approach, it appears natural to rely on component modeling to perform a verification phase in order to build safe systems. Thus we study how to ensure safeness for components whose models take into account time and uncertainty.

Nevertheless, software dependability cannot be proved by relying on a single technique. Some properties are decidable and they can be checked using formal methods at the model level. By contrast, some other properties are not decidable and they require non exhaustive methods such as abstract interpretation at the code level. Thus, a verification method to ensure generic component dependability must take into account several complementary verification techniques.